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Introduction 

In the book, Black Elk Speaks, when Black Elk 
told John Neidhardt about how the sacred pipe was 
brought to the Dakotas by White Buffalo Calf 
woman, he finishes his story by saying "this they 
tell, and whether it happened so or not I do not 
know; but if you think about it you can see that it 
is true." 

There are many things that we as Native people 
see as self-evident. Much of this is gained as 
cumulative knowledge within a people. However, 
it is often difficult to convince non-Natives of the 
truth of such beliefs and perspectives and we Na- 
tr % people end up getting sidetracked from believ- 
ing into trying to prove that which we believe in. 
The history of federal education policy is full of 
examples of changing beliefs about the best way to 
educate American Indian and Alaska Native stu- 
dents. In the early decades of this century, the 
Progressive education movement's influence led to 
a federal Native education policy which promoted 
retention of Native cultures and a multicultural 
approach to education. This policy was congruent; 
with a view of Native communities and govern* 
ments as worthy of recognition in their own right 
At mid-century, federal education policy had 
shifted to an assimilationist model and the goal of 
Native education was to get Native people into the 
American economic mainstream as quickly as pos- 
sible. Throughout these shifts, the belief of Native 
people has been that education should integrate 
goals of both cultural sustenance and economic 
self-sufficiency. From a non-Native perspective, 
however, these goals have been viewed as incom- 
patible. Using the term decentralization to denote 
efforts to ensure wider representation of and more 
responsiveness to legitimate interest groups (for 
our purposes, Native people), Weiler ( 1990) points 
out: 

Hie linkage between culture and learning 
tends to benefit from a more decentralized. 



disaggregated notion oflearnlng and educa- 
tional content.* 

...decentralizing the contexts and contents of 
learning as a means to recognize the diversity 
and importance of different cultural environ- 
ments In one society Is generally considered 
meaningful and valid. At the same time, 
however, it encounters the conflicting claims 
for a kind of learning that Is less geared to 
the specifics of cultural contexts and more to 
the national..* universalities of dealing with 
modern systems of technology and com- 
munication, (p. 439) 

The role that evaluation plays in determining 
federal education policy is also critical As Weiler 
notes: 

(Regarding] the relationship between 
decentralization and evaluation... I am argu- 
ing that this relationship is problematic for 
three different but interrelated reasons... 
These reasons are: (a) modern pluralist 
societies Increasingly face a lack of consen- 
sus on Uie objectives of education and hence 
on the criteria for evaluating the perfor- 
mance of educational systems; (b) there is.. .a 
very close linkage between evaluation and 
control, which is difficult to reconcile with 
the basic premises of decentralization; and 
(c) evaluation tends to be seen and used as 
a means of compensatory legitimation in its 
own right, that is, more for Its legitimating 
than for Its informative capacity, (p. 442) 

For ^iative people-students, parents, and 
Elders alike-evaluation studies have become 
tiresome exercises. Hence the "not another study" 
syndrome. Evaluation can provide dismal statis- 
tics about the failure of Native students to achieve 
as well as non-Native students. Or evaluation can 
provide information on how American schools fail 
their Native students. The choice of which evalua- 
tion model to use in determining federal education 
policy today is particularly important since most 
Native students are enrolled in public schools, 
which themselves are undergoing critical review 
and reform. 
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This paper was developed based on the review 
of data generated from the Indian Nations At Risk 
(INAJl) Task Force hearings across the country as 
well as on a literature review focused on evaluation 
studies either dealing specifically with the educa- 
tion of American Indian and Alaska Native stu- 
dents or which included Native students as part of 
the sample of students. The INAR hearing record 
itself is compelling and, as noted in the INAR 
summary -- Open Discussion with NACIE and 
Task Force Members, "the testimony from Indian 
people must be heard and taken in all sincerity" (p. 
3). 

The purpose of this paper is to present and 
discuss strategies for effective continuous evalua- 
tion of the education of Native students. Those 
strategies are intended as plans for systemic 
reform in order to establish ongoing evaluation 
systems. As a backdrop to this analysis, we need to 
first identify the systems responsible for delivery 
of educational services to Native students. These 
are: 

• The Federal government which is respon- 
sible for Bureau of Indian Affairs (BIA) 
funded schools through the Department of 
the Interior, as well as supplemental 
education programs funded through the 
Department of Education; 

• The state education agencies and local 
public school districts which now have 
responsibility for educating the majority of 
Native students; 

• Native governments which may operate 
their own schools, education programs, 
and/or tribal colleges; and which also have 
a responsibility to advocate for their con- 
stituents in dealing with other governmen- 
tal bodies. 

A primary task in this paper is to identify issues 
regarding the evaluation of the education of Native 
students. The hearing record of the INAR Task 
Force has noted that while thive are various in- 
dividual data collection/analysis efforts occurring 
at the national, state, tribal, and local levels, a 
national database on Native education is sorely 
lacking and must be established. This paper will 
also discuss what data and data collection systems 
do currently exist as well as how state and local 
education agencies and Native governments might 
coordinate data collection efforts in order to 
facilitate development of a national Native educa- 
tion database. When considering such a data base, 
it is also important that we (1) look at what 
measures, e.g., standardized test scores, are cur- 
rently being used to assess Native student achieve- 



ment (2) determine the appropriateness of these 
measures; and, if current measures are not ap- 
propriate, then (3) suggest potential systemic in- 
dicators of performance. 

Another task of this paper is to identify the 
various groups which "have a stake" in the con- 
tinuous evaluation of Native education. The 
evaluation needs of these various stakeholders can 
and do differ. We need to ensure that the data 
promoted by any evaluation are meaningful to all 
stakeholders. 

Finally, it is important to identify the elements 
of an evaluation model that would be responsive to 
the needs seen within Native education. Not all 
evaluation models contain mechanisms for yield- 
ing appropriate, practical and timely information. 
As witnessed in the INAR hearings, an evaluation 
model must offer Native people strategic informa- 
tion designed to promote success in education, not 
to simply document failure. 

Furthermore, the evaluation process must 
allow for participation of the various stakeholder 
groups. Finally, in recommending how to imple- 
ment strategies for continuous evaluation of Na- 
tive education, it is important to state these 
strategies as practical actions to be taken by each 
stakeholder group. 

This final point regarding practicality is central 
to our discussion and formulation of evaluation 
strategies. Whatever policies develop from the 
INAR report, they must focus on the greater im- 
practicality of keeping Native people underedu- 
cated and uninvolved in American's future. As 
Native peoples, we have much to give this nation 
in terms of cultural and social benefits. 

Evaluation and Federal 
Education Programs 

The role that evaluation plays in regard to 
federal education policy is an especially pertinent 
area to review here since much of Native educa- 
tion, as a distinct pedagogy, involves federal fund- 
ing. This is not to ignore the fact that public 
schools are the principal providers of educational 
services to the vast majority of Native students. 
However, as the testimony throughout the INAR 
hearings has evidenced, Nat * parents have very 
little input into policy matters at the local level 
although much of what happens regarding Native 
education is directly related to programmatic 
policy concerned with federally funded programs, 
like the Indian Education Act (ISA) formula grant 
programs, which are administered by public 
schools. 

House (1980) describes how federal educational 
evaluation policy evolved. In 1965, when Senator 
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Robert Kennedy pushed for an evaluation proviso 
to the Title I (compensatory education) program, 
this requirement was added to ensure tha v , schools 
use the new federal fluids to good advantage and 
thus, evaluation reporting was meant to assure 
that schools were responsive to parents 9 percep- 
tions about what their children needed. Unfor- 
tunately, the passage of the Elementary and 
Secondary Education Act, of which Title I was a 
part, also coincided with the introduction of Plan* 
ning Programming and Budgeting System (PPBS), 
a systems analysis mode which had been used 
extensive 1 ' in the Pentagon, and was now applied 
to a wider array of federal programs including 
education. Thus, while Kennedy's intent was to 
use evaluation to assure that schools would be 
responsive to poor parents, evaluation as eventual- 
ly implemented with Title I was used instead as a 
way to identify the most efficient approaches to 
educating disadvantaged students. Because of the 
evaluation requirements of the PPBS model, the 
implementation of Title I became an exercise in 
"developing programs that could be stated, 
measured, and evaluated in cost benefit terms' 1 (p. 
200). The model assumed that evaluation should 
be used to logically examine how a set of inputs, 
e.g. money or instructional strategies, was linked 
to a set of outputs, e.g. standardized test scores, 
and that this linkage was for the purpose of defin- 
ing the most efficient system of education service 
delivery. In other words, evaluation became 
viewed solely as a way to measure the production 
function of these federal education programs. 

The results of the first few years of Title I 
evaluations did not please the federal program 
staff under the Assistant Secretary for Program 
Evaluation (ASPE). They felt that the evaluations 
did not yield any common output measures to 
facilitate cost-benefit analysis. However, the 
evaluations, which were conducted locally, were 
almost all positive. ASPE then commissioned a 
study for the express purpose of examining what 
amount of resources yields what quantity of gain 
in a specific test score. That study, however, did 
not discover gains in test scores. Feeling that the 
problem was in the data supplied by the schools, 
ASPE then embarked on its own data collection 
and the Title I national surveys were commenced. 
However, the 1968 national survey showed no 
gains in test scores and the 1969 survey results 
were never made public (House, 1980). 

ASPE then moved to centralize more control 
over program inputs and design as well as evalua- 
tion. Furthermore, planned variation, i.e. the sys- 
temic introduction of different input and output 
measures, were introduced. The Follow Through 



program was one of the first federal education 
programs to undergo planned variation and to use 
other "experimental design" concepts. Specifical- 
ly, the use of control groups was introduced into 
the evaluation of the Follow Through program. 
The only evaluation information ASPE felt was 
worthy of consideration was that which focused on 
the production function, and the only evaluation 
models ASPE felt were worthy of consideration 
were those that used experimental methods and 
statistical techniques. The use of sponsors, i.e., 
specialists with special instructional models, was 
also introduced to the Follow Through program. 

From 1968 through 1977, work on the Follow 
Through program evaluation continued. The first 
evaluators, Stanford Research Institute (SRI) ini- 
tially stated that a broad array of evaluation 
criteria would be used including, among many 
other broader social measures, parental involve- 
ment/attitudes, comparisons of cognitive and affec- 
tive development, and responsiveness to 
low-income children and parents. 

Despite SRI's initial statements, however, from 
1968-1971, only cognitive tests were administered. 
Each year, annual Follow Through meetings 
erupted with protests from community groups and 
sponsors who were satisfied that the programs 
were successfully meeting local needs. These 
protests were directed against the evaluators' bias 
in looking solely at the production function and the 
evaluators' use of what community groups and 
sponsors considered biased indicators. After 
spending $12 million by 1971, the evaluations had 
found no positive Follow Through effects. Several 
ASPE staff resigned, new staff were appointed and 
a new evaluation directive was initiated. 

From 1972-73, while SRI continued the data 
collection, the Huron Institute was contracted to 
draw the evaluation sample, Abt Associates was 
contracted to conduct the data analysis, and a 
federally appointed panel was given the task of 
selecting the evaluation instruments. Two cogni- 
tive measures, the Metropolitan Achievement Test 
(MAT) and the Raven's Colored Progressive 
Matrices test were chosen and, despite the fact that 
the panel felt that no adequate noncognitive 
measures existed, two noncognitive tests were 
chosen for use, namely, the Intellectual Achieve- 
ment Responsibility Scale and the Coopersmith 
Self-esteem Inventory. 

The following year, the annual Follow Through 
meetings again erupted in discord. House (1980) 
reports that parents, feeling that they had no role 
in decision-making about the program, wrote: 

We are tired of others deciding when a pro- 
gram is not good' or 'good' for us based on 
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their concept of 'data 1 and their concept of 
what Is "wrong' with our children and what 
la needed to correct these "wrongs/ ...Hie 
burden of being able to measure the kind of 
program we want should be that of the ex- 
perts rather th?ji to design a program that 
they know how to measure, (p. 28) 

The sponsors also wrote a document in support 
of the parents 1 concerns. Although this flare-up 
was similar to the previous meeting! the earlier 
meeting had produced some immediate, if tem- 
porary, changes. This time there were no attempts 
to address the criticisms. The national evaluation 
of federal education programs was by now com- 
pletely isolated from stakeholder politics. This was 
the last annual general Follow Through meeting 
(House, 1980). 

Ti;e next round of evaluation battles in federal 
evaluation shifted to Head Start. From 1974-1977, 
the Head Start evaluation, designed by Abt As- 
sociates, also proceeded as "impact" studies, based 
on the assumption that the effect of a program can 
be judged by its impact on students' tests scores. 
Although no difference was found between Head 
Start and non-Head Start classes, the push for a 
broader array of social indicators or for the use of 
qualitative data in the evaluation of federal educa- 
tion programs was essentially quashed. In 1974, 
the top federal official in charge of education 
evaluation noted that "if we find that parents and 
kids are enthusiastic but the evidence is anecdo- 
tal.. .it would be irresponsible for us to tell other 
communities to consider the model" (Report of 
Fourth Annual Follow Through Working Con- 
ference, 1974, in House, 1980, p. 210). Thereafter, 
educational evaluation models at the federal level 
became focused mostly on efficiency concerns. 

All in all, both the evaluation policy at the 
national level, based on a sy terns analysis ap- 
proach, and its focus on efficiency concerns have 
been seriously challenged by educational re- 
searchers. The prospect of finding the most cost- 
effective or Efficient model for providing 
educational change is not as easy as identifying 
simple inputs and outputs. There are entirely tco 
many other variables that affect education. Fur- 
thermore, as the field of evaluation has developed, 
the reliance on experimental design concepts or on 
a hard science paradigm of evaluation has also 
been challenged. 

In terms of rendering moie useful, practical 
and/or true evaluation data, the use of more par- 
ticipatory models of education has been advocated 
by many evaluation researchers. These re- 
searchers include Yvonna Lincoln and Egon Guba, 
whose model for "naturalistic inquiry" is proposed 
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in this paper as an appropriate model for the con* 
tinuous evaluation of Native education. 

In analyzing what went wrong with the Follow 
Through evaluations! House (19S0) points out the 
discrepancy between the various stakeholders in* 
volved in the program. Two stakeholder groups, 
parents and sponsors, sought to push evaluation 
back to assessing the broad set of social goals (e.g., 
greater responsiveness on the part of the schools 
to the concerns of low-income parents) that Follow 
Through sought to accomplish. On the other hand, 
the other stakeholder group, the then Office of 
Education, sought to reduce the program outcomes 
to a few common quantitative measures in which 
comparisons of program sites could be made. 
Thus, the goal of the evaluation of the Follow 
Through program changed from focusing on deter* 
mining the overall effects of the program to only 
determining which model gave the most effect for 
a particular cost. This means-ends reasoning com* 
bined with the ends test score gains expected 
by the federal agency led to what is called technical 
rationality. This type of thinking assumes that 
"rationality consists of lining up clearly defined 
alternatives and choosing among them in terms of 
their effect on the particular objective one has in 
mind" (House, p. 212). 

The evaluations of many federally funded 
education programs fall into this type of 
rationality, when in fact the goals of the education 
programs may be much broader tjnan the measures 
themselves indicate. For example, the IEA for- 
mula grant program which was/is authorized to 
impel Native community control in partnerships 
with public school officials for Part A formula 
grants and meet the special educational and cul- 
turally related academic needs of Native students 
was evaluated in the early 1980's through an im- 
pact evaluation. Based on data from a sample of 
115 projects, this Impact Evaluation of IEA Part A 
Programs initially viewed standardized achieve- 
ment test data as a mqor indicator of the impact 
of the formula grant program on academic achieve- 
ment. However, after a thorough review of 
relevant research and discussions with testing/ 
evaluation experts, the evaluators (Development 
Associates) deemed as inappropriate the use of 
achievement test data as an impact measure. A 
mega-analysis of Indian student achievement test 
data from several studies over the preceding 40 
years had shown that there had steadily been a 
significant improvement in test scores since the 
late 1960's. Furthermore, by the time the "Part A" 
impact evaluation was conducted, test scores of 
Native students had already risen to their highest 
level in over a decade, although they were still 
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below the national norm. Thus, the evaluators 
could not directly attribute this progress to the IEA 
formula grant program. They did, however, note 
that the program "may [in part] have contributed 
to the increase" (National Evaluation of Indian 
Education Act Part A Projects, Department of 
Education, 1983). 

This impact study used a broad array of other 
impact measures, including school attendance, 
and student self-esteem and attitudes toward 
school. The study also concluded that nationally, 
school attendance of Native students no longer 
seemed to be a serious problem and that Native 
students' attitudes toward school and themselves 
were also quite positive. The report also claimed 
that Native parents were more involved than ever 
before in public schools and that parents seemed 
to be reasonably supportive of the education the 
public schools were providing their children. The 
report noted that the data also suggested that the 
climate in public schools with respect to Native 
children was considerably more benign than 
reported 15 years earlier. Except for drop out rates 
of Native students, the data showed marked 
change since before the IEA formula grant pro- 
gram began, although these impacts could not be 
directly attributable to this program or the other 
programs serving Native students. While noting 
that the overall objectives of Congress in enacting 
the Part A Program are being achieved, the study 
found that continuing problems existed Specifi- 
cally: 

(1) achievement test scores are not univer- 
sally positive; 

(2) while relations between the schools and 
Indian community are generally neutral to 
positive, this Is not true everywhere; 

(3) the sensitivity of school administrators 
continues tu be viewed as a problem by 
Indian parents and community leaders in up 
to 30% of the LEAs; and 

(4) test score results for some groups of 
Indians students are considerably below the 
norm. (Nichols, 1984) 

In general the study was not very conclusive. 
One criticism of the study that this author had was 
that the IEA legislation used a considerably ex- 
panded definition of "Indian/ 1 thus covering stu- 
dents who had, in many cases, not been included 
in the samples used for previous Native education 
studies. Thus, by including Native students (and 
parents) who were in many cases more mainstream 
than those in the earlier studies, the IEA program 
data could consequently result in a trend toward 
the norm that might easily be misinterpreted as 
improvement of student performance. This con- 
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cern, I felt, was not addressed adequately in the 
IEA Part Aimpact evaluation, nor was itaddressed 
or brought up in the INAR hearings. Nevertheless, 
I believe it is an issue which should be of significant 
concern in the continuous evaluation of Native 
education since it most definitely affects the con* 
gruency of evaluation findings about Native stu- 
dents over time. This concern is further 
substantiated by the recent 1990 census findings, 
which demonstrates that the federal government 
count of American Indians (including Aleut and 
Eskimos) has tripled since 1960, an increase that 
cannot simply be attributable to an increased Na- 
tive birth rate. While the growth rate remained 
low in states which traditionally have had a Native 
population (12.59 percent in South Dakota, 33 per* 
cent in Arizona), the growth in Native population 
has been considerable in states in which Native 
populations prior to 1960, were snail; to wit, in 
Alabama there was a 117.7 percent increase from 
1980 to 1990, only one decade (New York Times, 
March 5, 1991). 

The concern raised here is not with the 
legitimacy of the data or the legitimacy of those 
individuals who claim Indian heritage; rather the 
concern has to do with the need to take into con- 
sideration the changing nature of the sample(s) 
used in any database for the continuous evaluation 
of Native education. Caution must be exercised to 
ensure that valid conclusions are drawn from any 
data that might be generated in the future. 

Evaluation Issues 

A recent Time magazine essay explained how 
political "theorists of the emerging world" have 
started sorting the world into categories labeled 
Old Paradigm and New Paradigm, with the 1990's 
being viewed as the boundary at which one age 
transforms into another. A paradigm is an ex- 
ample, a model, that helps us compare things, as 
in past and present. To be able to process informa- 
tion and solve problems, one must operate unon a 
set of assumptions. Thus, the paradigm we use to 
come up with new solutions must operate upon new 
and different assumptions. An essential element 
of this new thinking is the making of lists of what 
is old and what is new. In looking for new ways to 
explain the world in order to analyze problems and 
seek solutions, this New Paradigm-Old Paradigm 
game considers "what works (New Paradigm) and 
what doesn't work anymore (Old Paradigm)." 

In this scheme of things, centralized 
bureaucracy and big government are the Old 
Paradigm. The New Paradigm, on the other hand, 
allows for programmatic flexibility, change and 
decentralization as well as access to decision- 
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making by the people affected. In this sense, the 
so-called New Paradigm in fact has commonalities 
with the thinking in the 1960's that gave rise to 
Community Action Programs meant to empower 
communities with previously little or no access to 
decision-making and power. 

This New/Old Paradigm metaphor is a way of 
thinking about change and looking at what works, 
particularly as a basis for making decisions. This 
approach is especially important as we consider 
what has worked and what has not worked with 
respect to the education of Native students. It is 
somewhat reassuring to hear the author of the New 
Paradigm term, James Pinkerton, an assistant to 
President Bush, voice the opinion that "the conven- 
tional wisdom around Washington is that nothing 
works. [However,] Americans don't believe it" 

Although much of the testimony from the 1NAR 
Task Force hearings focused on what has not 
worked for Native students, the hearings also 
provided information from concerned Native 
parents, Elders and educators about what (from 
their perspective) does work for themselves and 
their students. 

It is important to keep in mind as evaluation is 
discussed in this paper that we must start to look 
at Native education not in terms of the Old 
Paradigm (centralized government, one organizing 
idealogy, one big idea, one big solution) but rather 
in terms of the New Paradigm — decentralized 
actions and pragmatic multifaceted strategies to 
determine what does work, why it works and how 
to share this information. In this sense, it is im- 
perative that any recommended evaluation 
strategies focus not only on the national level, but 
also on how we can bring the very groups affected 
by Native education policies and programs — Na- 
tive communities, tribes, Elders, parents and stu- 
dents — into the evaluation process. 

A. Data Collection Efforts At The 
National Level Regarding Data 
Regarding Native Education 

Since the Meriam Report of 1928, the need for 
a national database on Native education has b*en 
pointed out in virtually every study related to 
Native education. For instance, the 1969 Congres- 
sional report, Indian Education: A National 
Tragedy — A National Challenge, specifically 
noted that "One problem in evaluating the success 
of Federal programs for the American Indian is the 
extraordinary inadequacy of the statistics data 
presently available . . . Without data, problems can- 
not be adequately understood or delineated and 
consequently are neglected" (p. 109). This report 



was issued over 20 years ago, yet its remarks 
remain disturbingly apropos for the 1990s. 

This 1969 report recommended that the onus 
for developing and maintaining data on Native 
education performance be placed upon the BIA. 
Indeed, it was recommended that the BIA monitor 
Native student performance not only at BIA- 
funded schools, but also "make periodic checks of 
Indian performance data in public schools, and 
that data be reported to local and state school 
authorities, the Indian tribes or communities af- 
fected, and the U.S. Office of Education" (p. 135). 
The report also recommended that the BIA "should 
require improved evaluation components at the 
State and local levels ... Some uniform data collec- 
tion techniques should be established ..." (p. 132). 
In other words, the need for a national database 
consisting of consistent data collected across states 
was again in 1969 deemed necessary. 

Eleven years later, in 1980, the policy recom- 
mendations of the Education Commission of the 
States (ECS) demonstrated that a national 
database on Native education was still seen as 
necessary and that it also continued to remain 
largely nonexistent. The ECS report recom- 
mended that: 

... state departments and boards of educa- 
tion and local boards recognize and consider 
the need to establish and maintain a stand- 
ardized, centralized data collection system 
on Indian education. This data would be 
collected by focal agencies and shared with 
Indian and non-Indian people Involved with 
the education of Indian children. [Indian 
Education, Education Commission of the 
States. 1980) 

The 1980 ECS report also recommended the 
creation, by Congress, of a "National Center for 
Indian Education" as a central point for the collec- 
tion of information, including statistical data, on 
Native education. It was suggested that the Center 
have capabilities to provide technical assistance to 
"tribes, legislators, education policy makers, and 
others ..." (Ibid., p. 33). 

In 1972, the National Advisory Council on 
Indian Education (NACIE) was established by 
specific legislation resulting from the 1969 study 
cited above. Given its mandate to oversee all 
federal education programs that benefit Native 
students, NACIE however, has not had sufficient 
opportunity nor resources for developing or main- 
taining an extensive database. In its 15th Annual 
Report to the United States Congress, Fiscal Year 
1988, NACIE reiterated the lack of adequate and 
consistent data to assist in fulfilling its oversight 
function. Specifically, noting that the respon- 
sibility for maintaining such a database should 
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logically be a part of the National Center for Educa- 
tion Statistics (NCES), NACIE pointed out: 

...the need to work with the NCES. and the 
need to collect data from state education 
agencies with available statistics on Indian 
students. The Council must begin this and 
other projects which will provide ...the types 
of Information needed to assess the educa- 
tional needs of Indian and Alaska Native 
people. However, It is clear that the Ad- 
ministration and Congress must assist in 
this effort by directing the [NCES] to gather 
the types of information needed and provid- 
ing NCES with the money to do the necessary 
surveys and by directing the Office of Indian 
Education [In the Department of Education] 
and [BIA] to gather Information from all of 
their grantees, contractors, and BIA- 
operated schools. (NACIE. 1988. p. 3C* 

In its FY 1989 report, the NACIE further noted 
that its staff had again had to compile data from 
various sources and would continue to do so "until 
the National Center for Education Statistics ... or 
some other responsible entity assumes the collec- 
tion of necessary information on [Native educa- 
tion]" (1989, p. 12). Noting the many agencies from 
which it had to elicit information, NACIE cau- 
tioned that these agencies "use different sampling 
methods for arriving at their computations, and 
comparing similar data from one agency with 
another is discouraged ..." (Ibid.). 

Based on NACIE's statements, it appears that 
the NCES has been unresponsive to the efforts of 
Native educators and policy recommendations 
citing the need for a national Native education 
database. Although, in its High School and Beyond 
(HS&B) survey, the NCES was able to cite Native 
students data regarding the sophomore "cohort 
drop-out rate" it was able to do so only because the 
HS&B survey elicited a great deal of background 
information on individual students, thus enabling 
a significant sample of Native students (Drop-out 
Rates in the United States: 1988, NCES, p. 25). 

Approximately 90 percent of Native students 
now attend public schools, which are directly 
provided Department of Education funds - 
$54,276,000 in FY 1990 - through a formula grant 
program specifically targeted only for Native stu- 
dents* Public schools on or near Indian reserva- 
tions receive additional funding through the 
Federal Impact Aid Program; for these schools, in 
FY 1989, this amounted to $239,355,638 in main- 
tenance and operations funds, and $7,681,000 in 
construction funds. In FY 1989, Native students 
in public schools benefitted from an additional 
$23,000,000 in funds under the Johnson O'Malley 
Program, which provides monies to states and 
school districts to provide services to Native stu- 



dents (Brescia, Commissioned Paper 4 of INAR 
Supplement Volume, 1991). The fact that so much 
federal funding is earmarked specifically for Na- 
tive students - not to mention that Native students 
in public schools comprise about five percent of the 
services recipients of the FY 1990 4.4 billion dollars 
in Chapter One funds - there is certainly a com- 
pelling reason for the inclusion of Native students 
as a special subsample in all NCES data collection 
efforts. Yet, despite the logic of such an undertak- 
ing, these data are still not collected as a matter of 
policy by NCES. 

Another study conducted by NCES, the Nation- 
al Education Longitudinal Study (NELS) of 1988, 
could have afforded an excellent opportunity to 
collect significant data on Native students. Never- 
theless, this study represented simply one more 
instance where Native education, while not ig- 
nored, was benignly neglected. For instance, while 
the NELS specifically oversampled for students of 
Hispanic or of Asian or Pacific Islander origins, it 
did not attempt to do so for Native students. Fur- 
thermore, while both "regular" public and private 
schools were included among schools sampled, BIA 
schools were specifically, for some reason, ex- 
cluded. The authors of the NELS:88 report, A 
Profile of the American Eighth Grader \ note that 
*NELS:88 is a powerful vehicle for looking at at- 
risk issues" (p. v). However, the sample of Native 
students is small; thus, the authors note that while 
for most of the sampled populations inferences can 
be made from the data which are generally reliable, 
such is not the case "when estimates are made for 
relatively small subpopulations, such as for 
American Indians (N=315)" (Appendix B-5). It 
should be noted that the INAR staff has looked at 
how a specific subsample of Native students could 
be incorporated into the NELS sample; however, 
clearly that would be costly at this juncture. 

In addition to the effort coordinated by the 
NACIE to collect national statistics on Native 
education from various federal agencies, the BIA 
began recently to report aggregate test scores and 
school performance data (it refers to them as 
'report cards' 1 ) on BIA-funded schools, i.e., BIA 
operated schools as well as those schools operated 
by tribes through either contracts or grants from 
BIA, A total of 36 out of 180 BIA-funded schools 
were reviewed during the 1989-90 school year. 
These report cards are issued to tribes and Indian 
parents and are intended to assist these con- 
stituencies make informed choices on which school 
students should attend. This effort also involves 
the collection of alternative measures among a few 
BIA-funded schools (this effort is discussed in more 
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detail in the section of this paper dealing with 
potential systemic indicators), 

Related to this effort, BIA has indicated it will 
establish a research and evaluation component in 
the Office of Indian Education Programs (OIEP) to 
measure the progress of "Indian students on In- 
dian lands" in meeting or exceeding national 
norms by the year 2000 (Department of the Inte- 
rior, Goal and Strategy for BIA Education, 1990). 
To monitor such progress, BIA will monitor each 
school every four years. The parameters of this 
effort are limited, however, since the monitoring 
will only account for those students in BIA-funded 
schools and not (1) those other Native students on 
or near reservations who attend public schools, 
much less (2) Native students who may be tribal 
members but live in non-rwal, non-reservation 
settings and attend nearby public schools •• that is, 
the majority of Native students. 

Given NCES' and BIA's stances, we are essen- 
tially left with only NACIE's effort to compile na- 
tional level data on Native education as it is 
generated from a variety of sources. However, 
since NACIE cannot effectively deal with this task 
without substantial additional resources, it is im- 
perative that NCES take on this data collection 
and analysis responsibility. The rationale of the 
special and unique relationship between Native 
governments and the U.S. government could cer- 
tainly serve as justification within the Department 
of Education to undertake this effort to recognize 
the special needs of Native students. 

B. Data Collection Efforts At The 
State And Local Levels Regarding 
Native Education 

The question of what data collection and re- 
search efforts are occurring at state and local levels 
was never directly asked by the INAR Task Force 
at the hearings. However, the testimony delivered 
by a broad cross-section of tribal, state and public 
school district educational personnel did include 
some valuable data that could presumably be use* 
ful forboth the U.S. Departments of Education and 
Interior. It seems clear, however, that there is very 
little effective coordination between state and Na- 
tive governments and public schools with respect 
to their efforts to collect the necessary data on 
Native student performance. Nevertheless certain 
inferences about data collection efforts can be 
made from the testimony at the INAR hearings as 
well as from other documentation and research 
collected by the INAR Task Force. 

In 1980, the following slates had the highest 
proportion of Native students to other students: 



Alaska (20.6 percent), Montana (9,8 percent), Ok- 
lahoma (9.1 percent), New Mexico (7.8 percent), 
South Dakota (7.2 percent), and Arizona (4.1 per- 
cent). Most of these states have some office within 
the state education agency which has respon- 
sibility for coordination with the Native com- 
munities (Boyer, 1983), INAR hearings were held 
in each of these states, with the exceptions of New 
Mexico and South Dakota. Additionally, hearings 
were held in California, Minnesota, North 
Carolina and Washington. 

With the exception of Montana and 
Washington, each of the states in which INAR 
hearings wer* held had at least one representative 
from the state education agency present testimony. 
With the exception of North Dakota which was 
adjacent to a state where hearings were held, no 
state in which hearings were not held sent repre- 
sentatives to the hearings. One can presume that 
this lack of greater state participation was in large 
part due to funding restrictions on out-of-state 
travel. It is noteworthy, however, that Alaska and 
Minnesota, in addition to the state education agen- 
cy representatives, had representatives from their 
respective state legislatures at the hearings. 

Those states represented at the hearings indi- 
cated that they collected some statistics on Native 
education although there were varying degrees of 
specificity among the witnesses as to the type of 
data collected and analyzed. In three* instances, 
local school district or tribal represer tatives also 
alluded to state statistics, namely fo • Wisconsin, 
New Mexico and South Dakota, but provided no 
substantive descriptions of the type or breadth of 
the statistical data available. 

Boyer ( 1983) states that "In addition to national 
school measures ... many states have assessments 
of their own ... [Many states — Iowa, California 
and Michigan are cited — require standardizad 
achievement tests at certain grades; and] more 
than thirty states now require competency tests' 1 
(pp. 29-30). In general, the states that were repre- 
sented at the hearings did not present any com- 
prehensive data on Native student performance 
that were collected and reported on a regular basis 
by the state. This is not to say that such a practice 
may not occur, only that the hearings did not bring 
out this point. 

In general, the information cited by the state 
education agency representatives pertained to 
specific, mostly recent, one-time studies of general 
state and/or Native education performance rather 
than to any on-going and institutionalized data 
collection efforts on Native students. For the most 
part these studies were either (1) one-time 
statewide assessments that included Natives as a 
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special population — such as Helping Schools Sue* 
cced At Helping All Children Learn (Alaska State 
Senate report, 1989) or (2) specific reports on Na- 
tive education problems and policy recommenda- 
tions — such as Our Children, Our Future 
(Minnesota Indian School Council report to the 
state legislature, 1989). It is noteworthy that these 
data collection efforts were usually instigated by 
specific state legislative actions. For example, 
Arizona, in 1983 and 1985, conducted a survey to 
help formulate an appropriate policy direction con- 
cerning the state's role in Indian education which 
resulted in A Working Document on Indian Educa- 
tion (1986). In 1988, North Carolina adopted an 
"Indian Education Policy" which led to the forma- 
tion of a state advisory council on Native education. 
Minnesota similarly has a "Comprehensive Plan 
for Indian Education" (INAR Regional Hearings 
Reports). 

One model for data collection is the TRACKS 
program in Montana. This program could serve as 
a potential model for other states since it repre- 
sents a comprehensive statewide effort to gather 
information on Native education from kindergar- 
ten through the 12th grade, as well as on higher 
education. This Montana project grew out of 
original research conducted by Dull Knife Com- 
munity College and the Northern Cheyenne 
Dropout Project. It is important to note the issues 
brought up by persons affiliated with these Mon- 
tana efforts, regarding the collection, at both the 
state and local/tribal levels, of Native student per- 
formance data. The following statement outlines 
the concerns and observations regarding data col- 
lection at the tribal/local level made by the director 
of the Northern Cheyenne project: 

Upon receiving funding from OERI for my 
study, I began a lengthy process to determine 
the exact Issues to address and data to col- 
lect I worked cooperaUvely with the three 
schools serving most of the students on this 
reservation ... Each school expressed dif- 
ferent concerns about the data collecting 
process, but they Anally agreed on who 
would collect the data and how to manage 
Issues of privacy. 

One critical aspect of my data collection 
process was the involvement of school per- 
sonnel. ... Their involvement was ... ber 1 
cial for themselves as they learned a grt 
deal about their own school and student 
and as a result are very interested In U % 
outcomes. I strongly urge that school pei 
sonnel at all levels be Involved In research 
endeavors. 

We collected kinds of Information that were 
suggested by other studies on school com- 
pletion and tried to determine whether stu- 



dents completed, transferred, or dropped out 
of schools. However, the definition of •drop 
out" Is complicated because some students 
do not finish high school and others attend 
elsewhere. Because of the high rate of trans- 
fer among local schools, there was an under- 
lying concern about transfer problems. We 
also collected data on student performance 
that Included grade point average — overall 
and by subject, standardized test scores, and 
percentile ranking, to allow a comparison of 
measures. We reviewed student charac- 
teristics such as the number of days missed 
In high school, discipline problems, in and 
out of school suspensions, and which 
schools attended. 

We collected Information on 698 students 
which represents three cohorts — the entire 
student population who would have 
graduated in 1987, 1988, and 1989, for all 
three schools. It Is not a sample. 

... Others [who testified] mentioned the 
need for good research but say this is too 
dlfllcult or too expensive. This Is not the case. 
The difficulty of this research comes from 
working with dlfllcult Issues and with con- 
cerns of confidentiality. 1 have been fortunate 
to work with ... a tribal community college 
... located In the middle of the reservation. I 
had college students working on every aspect 
of the project: data entry, data collection, 
analyzing, and wilting. Their input was es- 
sential to the research and they learned and 
contributed a lot 

Moreover, It is Important to conduct research 
at a local level. When you look at national 
studies like High School and Beyond, there 
are small representatives of Indians. It is 
Important for local studies to compare to the 
national studies, and critique whether they 
represent Indians well. It Is only with good 
local research that we can evaluate national 
studies and have an accurate perspective on 
their meaning. I recotnmend others to take 
on local research. 

Hie state is now interested in the project and 
its outcomes and has now Initiated a 
TRACKS system, but our experience shows 
it can be done successfully at the local level. 
(Carol Ward. INAR High Plains Regional 
Hearing. August 20, 1990) 

The director of the TRACKS program made the 
following observations about data collection issues 
at the state level: 

We need to put pressure on the Office of 
Public Instruction, universities, and tribes to 
get the informa* on we need on the status of 
Indian education. Access to Information 
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equals power. (Ellen Swaney, INAR High 
Plains Hearing, August 1990) 

In summary, the records of the INAR hearings 
bear out the fact that data collection efforts are to 
some degree (albeit at varying levels) occurring at 
the state, tribal and school district levels. Almost 
all individuals who testified were able to share at 
least some data on their constituent populations 
regarding educational performance. Based on the 
content of individual testimony, especially those 
from programs affiliated with universities, state 
universities seem to be able to collect and analyze 
information much more readily then other entities. 
Given this fact, perhaps special efforts should be 
made to coordinate efforts among universities, in- 
cluding tribal colleges, to lead this effort across 
states. 

With respect to native governments that have 
undertaken comprehensive data collection efforts 
on Native education, it would appear that those 
that have done so have clearly seen the link be- 
tween a good statistical demographic information 
base and economic development efforts. An ex- 
ample of tribal information gathering for this pur- 
pose are the Mississippi Band of Choctaw Indians' 
two publications, Choctaw Education (1979) and 
The Choctaw School Study (1985), both of which 
offer much information on methodological con- 
cerns at the local tribal level. 

Local Indian organizations, particularly in 
urban areas, also were able to provide extensive 
information on Native education performance. Ad- 
ditionally, regional educational laboratories, 
funded by the Department of Education, can assist 
in coordinating data collection efforts among 
tribes, public schools, and states. Sahme ( 1990) has 
devised a "preliminary flow chart" for how this 
process could be structured. 

The evidence collected through the INAR hear- 
ings suggests that data on Native education does 
exists to a greater degree than individual Native 
educators seem to believe. However, there is cur- 
rently littleeffortbeingmade amongtribes, or local 
school districts and states with respect to coordina- 
tion and consistency of data collection efforts. 
Moreover, there does not seem to be any interstate 
efforts for collecting such information. Further- 
more, there is a lack of agreement on whose respon- 
sibility it is to coordinate these efforts and also take 
responsibility for analyzing and reporting data 
from these diverse efforts. 

C. Standardized Tests And Native 
Student Performance 

Standardized testing can be simply defined as 
the use of tests which (1) have the same or 



equivalent items meant to assign numerical values 
to samples of behaviors; (2) are administered using 
uniform directions and scoring methods; and (3) 
are usually interpreted through the development 
of norms for the group for which the test was 
developed. 

There are generally four types of commercially 
available standardized tests which are most fre- 
quently used in educational placement and grad- 
ing. Aiken (1976) defines those as: 

• Achievement Tests. An achievement test 
is one that has been designed to measure 
the knowledge and skills accrued in a 
specific content area. The content areas 
are usually school subjects taught at a 
given grade level. Commercially available 
achievement tests are most frequently 
multiple choice, group-administered tests 
of content selected to apply to a wide range 
of school programs. 

• Aptitude tests An aptitude test is a test 
that has been designed to measure the 
capability of an individual to profit from 
instruction in a specific content area. An 
aptitude test is designed to measure skills, 
traits, and talents predictive of future per* 
formance in the area. 

• Ability tests. An ability test is one deigned 
to measure the capability of one to perform 
in a content area. The difference between 
an ability test and an aptitude test is one 
of status; present capability versus poten- 
tial capability. 

• Intelligence tests. An intelligence test is 
one designed to measure an individual's 
ability to reason and perform verbally. 
The IQ test is a generalized form of ap- 
titude test for scholastic work. 

At all of the INAR hearings, the issue of useful- 
ness and appropriateness of using scores from 
standardized tests, particularly achievement tests, 
for purposes of measuring Native students' 
academic performance elicited the most vociferous- 
ly negative testimony. Rehyner (INAR Supple- 
ment Volume, Com. Paper 8) discusses how some 
if the changes — such as a strong reliance on 
standardized achievement tests — in the American 
school reform movement of the 1980's have hurt 
Native students. Breschia and Fortune (1988) also 
criticize the misuse of standardized achievement 
tests. Specifically; 

... Achievement tests may be used in four 
ways to make decisions about [students:] as 
a survey of attainment in a content area, as 
a diagnostic instrument to identify the 
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strengths and weaknesses of an Individual 
in a content area, as a readiness Indicator to 
determine If an Individual has attained 
enough prerequisite material to continue 
study In a given content area, and as a 
performance test to estimate the degree of 
learning of a body of content defined by 
Instruction. 

Generally, when standardized tests are used 
with American Indian students (on the reser- 
vation or In settings with low levels of accul- 
turation) and produce invalid results, the 
testa usually produce lower or less desirable 
scores for the Indian test-taken These score 
variations are not really explained by pro- 
gram related factors nor correlates of test 
performance which are frequently found In 
other situations. 

In program-related decisions the underes- 
timation of Indian performance on ability 
tests may result in the development of an 
inefficient program design. UnderestimaUon 
on achievement tests may result in the 
demise or modification of what in reality is 
an effective program, (p. 2) 

In addition to these limitations to the use of 
standardized tests, several other factors can in- 
duce bias within such tests when used with Native 
students. These include language bias, whereby 
Native students, from non or substandard English 
prominent language families, may read questions 
or interpret an answer inaccurately. Native cul- 
tural values that do not stress competitive be- 
haviors also can produce bias. 

Despite concerns such as those stated aoov., 
standardized achievement test? are precisely tne 
measures that many states and public schools rely 
upon to gauge academic success. Unfortunately for 
Native students, who are already affected by the 
cultural, income-level, and/or linguistic biases of 
these tests, there is now also a movement for 
competency testing at the national level. The 
education-focused newspaper, Education Week, 
recently reported that Educate America, Inc., is 
calling for the development of a "national achieve- 
ment test for all high school seniors, [and plans] to 
ask Congress to fund it and make it mandatory for 
all students in public and private schools" (Educa- 
tion Week, February 6, 1991). Another article 
(Education Week, December 12, 1990) additionally 
reported that, two foundations, including the John 
D. and Catherine T. MacArthur Foundation, have 
provided grants to help launch work on a ' national 
examination system" based upon the recommenda- 
tions of groups such as the President's Educational 
Policy Advisory Committee. The article also notes 
that the National Center for Fair and Open Testing 
(commonly known as FairTest), the leading critic 
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of standardized testing, has called these moves "a 
step backward at a time when states and school 
districts are developing more complex methods of 
measuring student performance," 

That standardized teste are inherently cul- 
turally biased is generally agreed to based on the 
differences in performance across gender, racial, 
income, and 6thnic lines: "white" males tending to 
score the highest. Although not touted as an 
achievement test but rather as an aptitude test, 
the SAT is the most commonly cited example of 
bias in standardized testing. The statement and 
chart below discusses relative SAT test scores 
along gender and racial/ethnic lines; 

Many students face multiple biases on the 
SAT. Black females, for example, are placed 
in "double Jeopardy" by the testa minority 
and gender discrimination. Hispanic 
females, who score nearly 200 points lower 
than white males, face triple bias against 
their language, ethnicity, and gender. In 
every ethnic and racial group, females score 
much lower on the SAT than males, as shown 
in the following table. 



Average SAT Scores, 1988 









Differ- 




Female* 


Males 


ence 


Asian/ Pacific/ American 


903 


956 


53 


Black 


724 


756 


32 


Mexican American 


783 


840 


57 


Native American 


805 


852 


47 


Puerto Rlcan 


732 


788 


56 


White 


907 


965 


58 



To adjust for the SATs biases, some schools 
add points to women's and minority 
applicants' scores when they make admis- 
sion decisions. But this Is at best a stopgap 
measure. The only long-term solution Is to 
e i the*- overhaul the SAT to make It fair or stop 
using It (Weis. Beckworth and Schaeffer, 
1989. p. 18) 

Refer to Hillabrant, Romano and StangdNAR 
Supplement Volume, Commissioned Paper 2) for a 
detailed overview of Native student performance 
on the two most frequently used aptitude tests for 
college placements; these are the Scholastic ap- 
titude Test (SAT) and the American College Test- 
ing Program (ACT). 

At the St. Paul, Minnesota INAR hearing, the 
superintendent of the Red Lake Independent 
School District in Minnesota noted that " Too 
often Indian children have succeeded academically 
in spite of negative and hopeles predictions made 
by white educators from whit* hievement tests." 
Additionally, he cites a Lako^a Times newspaper 
article on a Ford Foundation-sponsored commis- 
sion finding that "All groups that score lower than 
whites on standardized tests fare better when 
judged by their school or job performance. Part of 
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the reason is that tests are blind to strengths that 
tests makers do not understand, and thus distort 
evaluation 11 (INAR Great Lakes Regional Hearing, 
pp. 46-47). 

Furthermore, the validity and use of stand* 
ardized tests for any students or teachers is ques- 
tioned by the broader educational community, 
Groups such as FairTest are questioning the uses 
of standardized testing for determining higher 
education admissions and teacher testing, as well 
as the public schools' use of these tests as accurate 
measures of academic performance (FairTest and 
New York State Public Interest Resource Center, 
1990). 

Not only in regard to their use as measures of 
general educational performance indicators have 
standardized tests been questioned. The INAR 
hearings also brought out the problem of using 
standardized tests to assess and diagnose educa- 
tional problems. Regarding their use in special 
education programs, Dr. Marilyn Johnson states: 

... the placement of children Is a challenge 
because the children are assessed with tests 
designed for a different population. With In- 
dian students, we are mostly dealing with 
blllnguallsm or limited English proficiency. 
We should not have children placed In spe- 
cial education if they do not properly belong 
there. (INAR Southwest Regional Hearing, p. 
35) 

Dr. Johnson also piesents and discusses some 
alternatives to currently used measures for special 
education assessments and placements in Com. 
Paper 16 of this supplement volume to the INAR 
report. 

Despite the fact that many of these and similar 
concerns have been voiced in the past, the BIA 
nevertheless relies heavily on standardized 
achievement test scores, specifically the California 
Achievement Test, for assessing their schools 9 im- 
provement. The most recent (1988) report on BIA- 
funded schools includes statistics on CAT scores, 
drop-out rates, attendance, enrollments and school 
expenditures. It is illuminating to note one Native 
educator's comments on the appropriateness of 
relying solely on such statistics: 

... As a former Bureau of Indian Affairs 
teacher. I am familiar with the national study 
the BIA put out In 1988 called Report on BIA 
Education: Excellence in Indian Education 
Through the Effective School Process. The 
Utle of the report Is rather misleading. The 
report has more to do with failures In Indian 
education than any kind of excellence. It 
contains page after page of dismal statistics 
concerning the educational achievement of 
Indian students, both In BIA schools and 
public schools. It Is full of statistics on high 



dropout rates, low scores on national stand- 
ardized tests, and poor academic achieve- 
ment in colleges, glvi?^ one the Impression 
that there are no successful Indian students 
anywhere. 

I am sure the Indian Nations at Risk Task 
Force will be presented with an abundance 
of these types of statistics. I am not suggest* 
lug that these types of statistics be Ignored; 
that would be foolish. If Indian students are 
dropping out of high school at a higher rate 
than other students, then, of course, that Is 
of deep concern for all of us Involved In 
Indian education. What I am suggesting Is 
that you not make these types of statistics so 
overriding that all else Is lost In your report 
on Indian education. (Robert Perea, INAR 
Southwest Regional Hearing) 

As Native educators have attempted to look at 
Native student performance, they have relied less 
on standardized achievement test scores and more 
on measures of student satisfaction with school, on 
drop-out rates for example. During the INAR 
hearings, drop-out rates were the performance 
measure most consistently cited. It should be 
noted, however, that one person testifying before 
the INAR Task Force suggested that we must shift 
from focusing on negative measures to using more 
positive perspectives and criteria: 

We must stop thinking of success as reduced 
drop-out rates and fewer suspensions and 
start thinking of success as high graduaUon 
rates and post-secondary enrollment (INAR 
Great Lakes Regional Hearing, p. 36) 

D. Current And Potential Systemic 
Indicators Of Performance 

The concern over the use of standardized tests 
for individual and school assessments is not voiced 
only by Native educators. There are also efforts in 
several states to make assessment more authentic 
and to develop indicators of authentic learning. 

The term authentic work is used by the Nation- 
al Center on Effective Secondary Schools (NCESS) 
to characterize tasks which are considered "mean- 
ingful, valuable, significant and worthy of one's 
efforts" (NCESS, 1990). The term encompasses 
indicators that are culminating performances 
showing not only what a student has achieved but 
also the day-to-day activities required to learn and 
prepare for the performance. 

Educators, in general, recognize that current 
conceptions of curricula and assessment, as well as 
their interaction, are fundamentally flawed. Cur- 
ricula are often viewed as sets of information (sets 
of truths) rather than demonstrable skills; similar- 
ly, assessment methods like teacher lecture/ques- 
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tion techniques, classroom quizzes, and stand- 
ardized tests rely on a narrow set of information 
that students are asked to recall and invoke, whole. 
Both of these conceptions focus primarily on lower 
rather than higher level thinking skills. Describ- 
ing the rationale behind NCESS research, Wiggins 
(1990) notes, 

curricula should be written around essential 
tasks to be mastered instead of 'doctrines' to 
be learned... [Furthermore] a verbal and pas- 
sive view of knowledge leads us to falsely 
believe that.. assessment Involves a quasi- 
secretive after-the-fact sampling of the 
students' verbalized knowledge of the 'basic* 
facta, instead of a non-secret process of 
meeting known standards, while using im- 
portant and diverse facts. (NCESS, p. 10} 

Educational researchers like those at the 
NCESS view the current problems such as non- 
engagement of students in the classroom as related 
to the lack of authentic work being required of 
students: 

Authentic work is not about being tested 
after the fact. .it Is about being tested by a 
standard-revealing and important set of 
tasks.. .Most students can produce quality 
when It is expected: and when tasks, criteria 
and standards are engaging and demysUfled. 
(Wiggins, p. 11) 

If these criticisms of current schooling and as* 
sessment are true for non-Native students, they 
are doubly appropriate for Native students whose 
many tribal/cultural traditions vary from those of 
mainstream society. As Swisher ( 1990). in an over- 
view of research into Native students' learning 
styles notes, among Oglala Sioux, Yaqui and 
Navajo learners, 

...observation, self-testing in private, and 
then demonstraUon of a task for approval 
were essential steps In learning and that 
learning through public mistakes is not a 
valued method.. .This style of learning sug- 
gests a respect for an individual's ability to 
learn from their world experientially without 
constant supervision and correction from 
another Individual. It expresses a certain 
degree rf confidence in the autonomy of the 
individual to know when performance of a 
task is ready for public scrutiny, (p. 3) 

Swisher, as well as this author, cautions 
against the positing of a single Native learning 
style without further investigation, noting that 
this may further stereotype Native student be- 
havior and 'result in practice which is dis- 
criminatory or making inappropriate excuses for 
failure in teaching' 1 (p. 6). While several presen- 
ters at the INAR hearings commented on current 
teachers 9 ability or desire to teach to Native learn- 



ing style(s), this author believes that the criticism 
of current teaching and assessment methods as 
being unresponsive to different learning styles is 
equally valid for both non-Native and Native stu- 
dents. 

In response to rising criticism about its heavy 
reliance on the CAT scores, the BIAhas started to 
look at alternative indicators of performance. The 
BIA's Bureau of Effective Schools Teams (BEST) 
initiative has developed alternative student per- 
formance measures-input a6 well as output 
measures-some of which could serve as potential 
systemic indicators in the evaluation of Native 
education. The BEST initiative represents a spe- 
cial effort to improve BIA-funded schools. Cur- 
rently 41 out of 180, or 23 percent, of schools are 
designated as BEST schools. The Alternative 
measures include, among others: 

• Criterion-referenced tests (or teacher- 
made tests); 

• Portfolios of student progress, e.g., writing 
samples; 

• Non-academic participation (i.e., extra- 
curricular) rates; 

• Attendance rates; 

• Increased graduation rates; 

• Rates of decreased vandalism by students; 

• Increased holding power of a school to keep 
students and staff; 

• Implementation of new curriculum initia- 
tives; 

• Increased participation of parents and 
community members; 

• Increases in the variation of extra-cur- 
ricular activities offered by schools and 
participated in by students; 

• Improvements in staff development 
programs; 

• Implementation of written school improve- 
ment plans; and 

• Facilities improvements. 

While several of these indicators (e.g., perfor- 
mance on criterion-referenced tests) may not easily 
serve as comparative systemic performance in- 
dicators across several schools, certainly others 
such as increased graduation rates, improved at- 
tendance and retention (holding power) rates could 
easily serve as indicators of performance among 
different schools serving Native students. With a 
little more effort, measures such as portfolios of 
Native student progress, perhaps compared across 
a sample of schools with Native students, could 
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also serve as more dependable, appropriate and 
useful measures of systemic performance. 

As the need for more accurate assessments of 
student learning is acknowledged, potential sys- 
temic indicators of performance are being 
developed nationwide, not only in Native educa- 
tion. For instance, the state of Connecticut (along 
with six other states « Michigan, Minnesota, New 
York, Texas, Vermont and Wisconsin) is involved 
in developing alternatives to the use of stand* 
ardized tests in assessing student outcomes. In 
Connecticut, this effort has required the develop- 
ment of a "Common Core of Learning" for all stu- 
dents in the state, i.e., a comprehensive set of 
knowledge and traits against which students are 
individually assessed. The Connecticut model is 
based on a comprehensive review of reports/recom- 
mendations made by several agencies and or- 
ganizations such as the National Science 
Foundation, the National Academy of Science, the 
National Council of Teachers of Mathematics, and 
the American Association for the Advancement of 
Sciences. Recognizing that an overreliance on 
multiple-choice testing focuses on lower level 
thinking skills such as recall/recognition, the alter- 
natives emphasize "conceptual understandings, 
multiple representations and connections," among 
other mastery criteria. Students are then perfor- 
mance tested, i.e., they must demonstrate mastery 
of specific skills. For example, in sciences, stu- 
dents must demonstrate the use of various science 
apparati (e.g., a triple beam balance, graduated 
cylinders, a microscope and an electrical circuit) 
and design and execute a science experiment. In 
the language arts, students must produce a direct 
writing sample as well as be able to take notes and 
use those notes to answer listening comprehension 
questions in response to tape-recorded messages 
(Baron, 1989). 

Moreover, the National Commission on the 
Skills of the American Work Force (NCSAWF) has 
made recommendations regarding assessments 
specifically for demonstrated mastery of skills and 
subject areas. Several states are now incorporat- 
ing or considering the NCSAWF recommendations 
{Education Week, June 20, 1990). These recom- 
mendations include the abolishment of the Car- 
negie unit, which is also among the reforms 
suggested by some Native educators and others 
testifying at the INAR hearing. One native 
educator (Blanchard, 1990) goes so far as to recom- 
mend the abolishment of the grade structure. 

Clearly, if those individuals involved in Native 
education (1) examine what the current systemic 
indicators are telling Native students, parents and 
teachers, and (2) determine whether this is useful 



information f we can agree that current systemic 
indicators, such as standardized tests, absen- 
teeism and drop-out rates are not particularly use- 
ful in assessing the actual knowledge, skills and 
learning processes acquired by Native students. 
Any plans for further evaluation of Native educa- 
tion must not rely heavily on measures that may 
sometimes be outdated or inappropriate (such as 
standardized achievement testing)* Rather, the 
plans must allow for the use of alternative perfor- 
mance indicators such as those used by the BIA's 
BEST initiative or those being developed by the 
state of Connecticut and the six other states in the 
Coalition of Essential Schools. 

Much of the testimony from the INAR hearings 
seem to indicate that current systemic perfor- 
mance measures are not useful. As one INAR Task 
Force member asks: 

How can we circumvent an evaluation 
process that measures performance related 
to tightly specified academic criteria that is 
not only irrelevant to our children's view of 
the world, but which documents failure In- 
stead of relationships between students and 
teachers, resources and students, and so on, 
that contribute to failure? (Hill, Pedagogy 
and Self Determination, INAR ancillary docu- 
ment 1990) 

Stakeholder Groups In Native 
Education 

Native education involves several systems of 
service providers: federal agencies, state and local 
public schools, and Native governments. Addition- 
ally, each of these systems has categorical distinc- 
tions within itself. For example, the federal 
service providers include (1) the BIA-funded 
schools, e.g., those operated by the BIA as well as 
those operated by Native governments; and (2) the 
various ED programs which interact both with 
public schools directly (e.g., IEA formula grant 
projects) and tribes (e.g., vocational education, 
library services, and IEA adult education projects). 
Native education involves a complex network of 
interrelated agencies and responsibilities. Fur- 
thermore, these relationships are not unidirection- 
al as they involve several points for input from the 
consumers of educational services. Also, several 
other functions involve monitoring the delivery as 
well as impacts of services. 

All of the parties involved in Native education, 
from providers (e.g., schools, states, federal agen- 
cies, Native governments, Elders) to consumers 
(e.g., Native students, parents and governments) 
to monitors (e.g., Native parents, and govern- 
ments, Elders, states) have an interest — a "stake" 
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in ensuring that their concerns and perspectives 
are represented in the continuous evaluation of 
Native education. As noted in the INAR hearings, 
these stakeholder groups have different evaluation 
needs in terms of data, and require different pro* 
cedures and reporting formats to find the evalua- 
tion useful or practical. 

As primary stakeholders in the evaluation of 
Native education and as educational services con- 
sumers, parents realize the most immediate needs 
for effective evaluative information. At each IN AR 
regional hearing, and at many of the INAR Joint 
Issues Sessions at the 1990 National Indian 
Education Association (NIEA) Annual Conference, 
Native parents voiced their concerns about the 
quality of educational services for Native students 
and the effects/goals of educational institutions. 

The remarks made by one parent can be 
generalized across many others who expressed dis- 
satisfaction about currently used student perfor- 
mance indicators: 

- ...my son was tested with the CTBS test 
administered through the Bureau, and he 
scored real low. TTiey told him he had to be 
retained. He was In sixth grade and couldn't 
go on to seventh grade. As a parent, they 
wanted me to sign off on this. Usually the 
teachers and administrators call parents in 
to discuss a problem and ask the parent to 
sign off on a decision without giving the 
parent an option. Well. I said I didn't want 
to sign It. I said, "I know my child is better 
than that" And I said, The test determines 
he can't read or write, but I know his poten- 
tial." And so he therefore went ahead. I 
promoted him myself. The next year he was 
an honor student. He had real good grades 
all year, and I was glad I didn't sign those 
papers and have him retained. He is going 
to be k jaduating this year and thinking of 
going to college. (INAR Summary: "Special 
Session for Students and Elders ..." p. 5) 

That same parent felt that (1) many parents 
were intimidated by schools or (2) school 
administrators' expectations were low about how 
parents can assist their children. The same parent 
also indicated that parents have "common sense 
an i a desire to learn about what their children are 
learning, and from that base they will begin to help 
their children get educated and become better 
prepared." 

Evaluation strategies that focus only upon the 
compilation of statistics detailing the education 
failure of students to meet some elusive national 
norm are at best interesting for Native parents, at 
worst, depressing and debilitating. As noted in a 
previous section, parents of Native students who 
attend BIA schools, may receive "report cards" 



including aggregate school-wide test data on the 
performance of Native students. However, most of 
these parents cannot use these data to, as the BIA 
hopes, make informed choices about the schools 
their childrm should attend. In many cases, there 
is in fact no choice, or the choices are even less 
attractive than the schools in which the students 
are already enrolled. Parents would rather re; dive 
information about why those schools their students 
do attend fail to raise the students' performance 
levels. Knowing why schools « not Native students 
- fail would help parents make informed decisions 
on how to change the schools. 

Obversely, parents of Native students in public 
schools often have problems getting statistical in- 
formation on Native students as a group from their 
respective school districts. Thus, they are often 
unable to tell how Native students, as a group, 
compare with other students in their schools. Ac- 
cess to data-either aggregated for Native stu- 
dents, or comparative Native and non-Native 
data-(such as drop-out rates, absenteeism, reten- 
tion, graduation rates as well as aggregate Native 
standardized test scores) is critical for determining 
needs of students eligible for Indian Education Act 
(IEA) formula grant projects. School districts often 
cite confidentiality concerns when denying access 
to these data. However, parents, particularly 
those that serve on IEA project parent committees, 
need such data to fulfill their responsibilities. 
Those observations are made by this author based 
on ten years experience with IEA projects. 

In addition to student performance data, Na- 
tive parents need information on educational 
programs that are effective with disadvantaged, 
low-income, multi-cultural and/or bilingual stu- 
dents, i.e., categories into which many Native stu- 
dents fall. Of course, information specifically on 
successful Native education programs is needed by 
Native parents in order for them to replicate and/or 
then adapt in their own settings. 

Throughout the INARhearings, Native parents 
and Elders have called for schools to take a lead 
role in ensuring the survival of Native cultures and 
languages. Indeed this philosophy reflects the spe- 
cial Native goals added by the INAR Task Force to 
the national goals for the year 2000. In line with 
this, any continuous evaluation strategies for Na- 
tive education should allow for data collection, 
analyses and reporting on special programs that 
effectively sustain and develop Native cultures and 
languages. 

Native Elders, as parents and grandparents, 
expressed their concerns throughout the INAR 
hearings. These concerns centered around the 
desire to see the sustenance of Native cultures and 
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languages as a primary goal of the education of 
Native students. As a specific stakeholder group, 
in the continuous evaluation of Native education, 
the involvement of Native Elders must serve as a 
touchstone in planning and implementation of fu- 
ture evaluations. Many individuals testifying at 
the INAR hearings, expressed concern over the 
realization of the tremendous fragility of Native 
cultures and languages. Further evaluations in 
Native education must explore particularly effec- 
tive programs that nurture the ongoing develop- 
ment of Native cultures and languages. 
Furthermore, mechanisms for sharinginformation 
about these programs must be developed and for* 
malized. Presently, no one source of information, 
on effective Native culture and language sus- 
tenance programs, exists. Coordination efforts 
among Native governments must be promoted by 
federal agencies such as BIA as well as by national 
organizations such as the National Indian Educa- 
tion Association and the National Congress of 
American Indians. It is hoped that recent Con- 
gressional efforts, such as the Native Languages 
Preservation Act, will facilitate such coordination 
and development. 

Native governments as both providers and con- 
sumers of Native education have distinct needs in 
terms of information that may come out of the 
continuous evaluation of Native education. Closer 
coordination between state education agencies and 
Native governments can lead to the easy retrieval 
of Native student performance data. However, 
state and local school districts must first realize 
their special obligation to Native students. Other 
commissioned papers (e.g., papers 1, 9, and 20) in 
this INAR Supplement Volume offer descriptive 
information on the obstacles put up by states in 
recognizing this need. Primary among these 
obstacles is the attitude, based on years of civil 
rights issues, that Native students are like other 
minority groups; therefore, to treat Native stu- 
dents differently would lead to charges of dis- 
crimination from these other groups. It behooves 
Native governments and oflf-reservation Native 
communities to take the lead in making states 
realize that the status of Native people is unique. 
The current efforts in Minnesota can serve as a 
model for Native governments and communities to 
use in working with their respective states (Our 
Children, Our Future, Minnesota Indian School 
Council, 1989). 

The needs of other stakeholder groups that 
represent specific educator subgroups in Native 
education are well stated in other commissioned 
papers of this INAR Supplement Volume (e.g., 
papers 5, 7, 12, 13, 14, 15, 17, 18, and 19). Further- 



more, Com. Paper 18 provides insight into how 
indicators of institutional success and student per- 
formance outcomes differ at tribal colleges from 
those at non-Native institutions of higher educa- 
tion. 

Strategies for Continuous 
Evaluation of Native Education 

The federal government has sponsored 
numerous evaluations, task force studies, policy 
review reports etc., on the education of Native 
students, both young and adult. The INAR hear- 
ings have generated many statistics on specific 
programs (in public schools as well as Native-con- 
trolled or tribal schools) and their impact on Native 
students. The state education representatives 
from several states with large Native populations 
have reported on their monitoring of Native educa- 
tional performance. By examining the hearing 
record data, other information collected by the 
INAR Task Force, and their own management 
information systems, the Department of Education 
and BIA potentially have information to make 
better informed decisions about the extent of Na- 
tive education services and more importantly, the 
quality of services. 

To a reasonable degree, decisions about these 
programs can be based on the past performance of 
these programs. Some changes in the reporting of 
outcomes data can assist in judging program per- 
formance. As noted by one INAR Task Force mem- 
ber, "...in addition to a drop-out rate for [Native] 
students that exceeds 65 percent, and a post secon- 
dary drop-out rate that is estimated at 75-93 per- 
cent, hundreds of students effectively drop-out of 
school and physically never miss a day ... the 
statistics are very clear that we can out disad- 
vantage all groups" (Hill, 1990). 

It is imperative that strategies be developed to 
take evaluation of Native education beyond the 
discrepancy-based evaluation models concerned 
mostly with the collection of statistics on the lack 
of Native student achievement (e.g., low stand- 
ardized achievement test scores). The collection of 
comparative statistical data on Native students is 
important to draw some conclusions about how 
well schools are serving Native students in relation 
to non-Native students. However, it is important 
to make certain that these data are valid and 
appropriate. Certain systemic indicators like 
dropout and absenteeism rates are likely more 
valid as measures of the schools 1 responsiveness to 
Native students and parents concerns. Com- 
parisons of these indicators across the three 
categories of school systems (federal, state/local, 
Native-operated) along with descriptive informa- 
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tion on why these rates may vary (e.g., different 
policies, school atmosphere, expectations) can give 
Native students and parents useful information. 
The collection of statistical data - using various 
measures which can yield comparative informa- 
tion about the performance of Native students rela- 
tive to that of other groups - should be an ongoing 
function of state education agencies and also of 
NCES. 

In terms of data collection for discrepancy 
evaluation, the TRACKS program in Montana as 
well as other efforts such as those of the Con- 
federated Tribes of Warm Springs Reservation in 
Oregon can provide direction to Native educators 
and federal education officials alike. The Warm 
Springs model (Sahme, 1990^ calls for Native 
educators to promote the establishment of Indian 
education standards for all Indian education 
programs. Noting that BIA has standards for its 
schools, Sahme calls for tribes to take part in 
setting standards for the schools they run and to 
work with public schools to set performance stand- 
ards and jointly devise methods for achieving these 
standards. Furthermore, he notes that public 
schools receiving IEAformula grant funds must be 
held responsible for also setting long-range goals 
for meeting standards that would be jointly 
developed by the Native parent committees and 
the schools. These standards would be above and 
beyond the objectives of the funded project, and 
would extend across the school curriculum and 
programs. This effort could be coordinated 
through the seven regional education laboratories 
in conjunction with the IEA Indian Technical As- 
sistance Centers funded by the Department of 
Education (Sahme, 1990). 

More important in the evaluation of Native 
education is finding out what is working and just 
as importantly why it is working. In the language 
of the Old-New Paradigm, we must go beyond the 
big picture of national failure and refocus on snap- 
shots of localized success in order to assess how and 
why certain approaches work. These snapshots can 
focus on regional efforts or particular stakeholder 
group claims, concerns or issues. The Department 
of Education and BIA must redirect their organiza- 
tional efforts at increasing the knowledge base in 
those areas where little research and evaluation 
has thus far been conducted. For example, we not 
only need to know the drop-out rates in our schools, 
but also why some Native students drop out and 
others don't. 

A The Role of Naturalistic Inquiry 

The INAR hearing record clearly demonstrates 
that the current evaluation models, based mostly 



on learning deficits, have not and do not serve 
Native education well, at least in terms of finding 
solutions to Native education problems. These 
models only inform us of the tremendously high 
Native student drop-out rates, the poor perfor- 
mance of Native students on standardized tests, 
and the ever increasing tardiness and absenteeism 
among Native students attending either BIA- 
funded or public schools. One of the most salient 
drawbacks to these models is that they fail to give 
us the underlying reasons for these dismal statis- 
tics. 

Paul (in the INAR Supplement Volume, Com- 
missioned Paper 7) cautions that we must move 
beyond the hard science paradigm in Native educa- 
tion and move toward an evaluation model which 
will empower the participants - in other words, 
one that "would become a part of the growth and 
development of the children, families and [schools] 
within the communities (p. 16)." Charleston (1990) 
discusses an evaluation framework for the Indian 
Health Service (IHS) which would assist both the 
health care providers as well as the health care 
consumers — i.e., tribes — make better informed 
decisions. He also presents an emergent paradigm 
of evaluation, one with the intent of facilitating an 
understanding of the organization in relation to its 
environment and the complex network of relation- 
ships. This model, based on principles of naturalis- 
tic inquiry, could also be adopted to Native 
education and employed in future evaluations of 
Native education. 

Naturalistic inquiry, or fourth generation 
evaluation, evolved from earlier models of social 
and educational research. The application of 
naturalistic inquiry can assist future evaluation of 
Native education move beyond the clash of evalua- 
tion goals - similar to those documented by House 
(1980) in regard to the evaluation of three federal 
evaluation programs in the 1960's and 1970's - 
which are present in Native education as docu- 
mented in the INAR hearings. These distinct goals 
were referenced in the introduction of this paper, 
but need reiteration here. Namely, federal goals 
for Native education (as reflected and imple- 
mented by public and BIA schools) are mainly 
assimilationist and primarily concerned with en- 
suring the most efficient means of bringing Native 
students to the performance levels of non-Native 
students. Hence evaluation strategies have 
focused on comparisons of Native and non-Native 
student performance utilizing outcome measures 
that could be applied across disparate populations. 
Past evaluations and policy studies have thus been 
based solely on discrepancy evaluation models, 
with little or no analysis of the systemic factors 
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that affect/result in these outcomes. Nor have past 
evaluations examined Native education as the 
complex network/system that it is; rather, past 
evaluations tended to focus on categorical distinc- 
tions such as specific federal programs or service 
agencies. 

Native educators and stakeholder groups , on 
the other hand, see the goals of Native education 
as being culturally distinct from mainstream 
society while allowing for simultaneous participa- 
tion in mainstream society. Native parents want 
their children to be educated to succeed in school 
but also to learn in culturally appropriate ways and 
to sustain/preserve their distinct cultures and lan- 
guages. For them, evaluation of Native education 
must not only detail how schools fail their children 
in reaching the goals of Native people, but also 
provide information on how they as Native 
parents, Elders and educators can work to rectify 
any shortcomings related to educating Native stu- 
dents. Furthermore, from the perspective of Native 
communities, Native education is a singular but 
complex web of services, providers, and programs, 
e.g., BIA and/or public schools, JOM, Title V. This 
web is a singular whole which cannot be dealt with 
program by program, since each Native student is 
impacted by all entities. In short, there are multi- 
ple realities at work Native education which must 
be taken into account in evaluation. 

The principles of naturalistic inquiry, 
developed by Yvonna Lincoln and Egon Guba, are 
responsive to the complexity and multiple goals 
within an organization. If Native education were 
viewed as a complex organism, then these same 
principles would certainly meet Paul's call for an 
"empowerment" model for the continuous evalua- 
tion of Native education. Naturalistic evaluation 
provides information in terms of naturalistic 
generalizations: for example, like the non-proposi- 
tional, qualitative information garnished from 
reading a novel. Naturalistic inquiry uses ordinary 
language and is aimed at non-technical audiences 
like teachers or the public. It is based on "informal 
everyday reasoning* and tries to "understand the 
everyday world in the experience of those who live 
it." The case study is an example of naturalistic 
evaluation (House, 1980, p. 279). 

Furthermore, naturalistic evaluation recog- 
nizes that there are competing claims, concerns, 
and issues to be considered which make it neces- 
sary to arrive at findings useful to the various 
groups that have a stake in the evaluation (Char- 
leston, 1990). 

Naturalistic inquiry (Guba and Lincoln, 1985) 
builds upon the work of Schwartz and Ogilvy 
(1979) which analyzes and reports the emergence 



of new concepts in a variety of fields and disciplines 
including physics, chemistry, evolution, mathe- 
matics, philosophy, psychology, and educational 
research. These emerging concepts basically 
define the world in a much different way than has 
traditionally been the case. In their analysis, 
Schwartz and Ogilvy have abstracted seven areas 
that characterize how these new concepts, in what 
they call the new or emergent paradigm, differ 
from the old or dominant paradigm. The pre- 
viously mentioned Old/New Paradigm ideas 
regarding political organization and government, 
represent a special application of this much 
broader emergent paradigm. The seven charac- 
teristics of Schwartz and Ogilvy's emergent 
paradigm, very simplistically stated, are: 

(1) Complexity: Reality is diverse, complex 
and interactive and is becoming increas- 
ingly so; therefore, it is impossible to 
separate out any one thing from it's inter- 
active environment and study it in isola- 
tion. Furthermore, systems can't be 
viewed simply as the sums of their parts; 
as they become more complex they develop 
their own unique qualities which cannot be 
accounted for from their parts. 

(2) Heterarchy: Reality or nature, rather 
than ordered in terms of hierarchy or logi- 
cal order, is ruled by many interacting (and 
rapidly shifting) factors. Furthermore, 
order may only exist in human thinking 
about the world, and may not in fact be real 
at, all. 

(3) Holography: This is the hardest concept to 
grasp since it is based on metaphor. Simp- 
ly put, the old concepts of the world were 
mechanical; the universe (and reality) 
operated like a clock. The emergent 
paradigm views reality as more like a 
hologram. A hologram is an image 
produced by breaking an image into 
several patterns of light from several dif- 
ferent perspectives. Furthermore, each 
piece of the holograph can reproduce the 
whole image if all others are destroyed. 

(4) Indeterminacy: The behavior in a 
mechanical world view can predict the fu- 
ture if all variables are accounted for, i.e., 
this is a determinant view of reality. The 
emergent paradigm views the world as in- 
determinate, i.e., unpredictable; one can 
never account for all possible variables. 
The future states of systems are thus un- 
predictable. 
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(5) Mutual Causality: The mechanical model 
of the old paradigm views the world as one 
where causality is linear. Therefore, in the 
old paradigm it was important to learn 
what caused what. On the other hand, in 
the emergent paradigm, causality is 
material and evolving; the distinction be- 
tween cause and effect in a system is mean- 
ingless. 

(6) Morphogenesis: In the old paradigm, a 
system was constructed or assembled from 
parts organized in a knowable plan. In the 
emergent paradigm, systems are evolving 
in morphogenetic (organic) ways which are 
unpredictable and spontaneous under con- 
ditions of diversity, openness, and mutual 
causality* 

(7) Perspective: In the old paradigm, objec- 
tivity was valued. The emergent paradigm 
states that objectivity is impossible; e.g., 
instruments and processes are not neutral, 
they alter reality. However, in place of 
subjectivity as the alternative to objec- 
tivity, the emergent paradigm suggests 
perspective as a more useful concept. In 
evaluation, one cannot be objective, since 
the observer changes the nature of what is 
being studied. Rather, one must try to 
present as many perspectives as possible 
to ensure that the evaluation is true. 

[Recognizing that this is a very simplistic ex- 
planation of the very complex concepts underlying 
the emergent paradigm, this author refers any one 
interested in this new paradigm to the works of 
Guba and Lincoln as well as to the original 
monograph by Schwartz and Ogilvy. Both are 
cited in the references.] 

The emerging way of viewing the world, and 
approaching evaluation, is more in line with tradi- 
tional Native beliefs about the world/reality. That 
is to say, Native peoples view the world as complex, 
inter-connected in non-linear relationships 
(heterarchic), dynamic, unknowable (indeter- 
minant), changing/movingin several simultaneous 
cycles (mutual causality), growing as a whole (mor- 
phogenesis) and consisting of many perspectives. 

Given the extent to which past evaluations of 
Native education have relied on old paradigm con- 
cepts, it is important to restate how the world of 
Native education actually is perceived in terms of 
the emergent paradigm characteristics. 

Complexity: Native e^ 1 cation cannot be simp- 
ly described in terms of distinct systems of educa- 
tion (e.g., BIA schools, public schools, federal 
programs, Native government operated schools). 
Rather Native education is a complex inter- 



relationship of systems at the national, state/local, 
and Native community levels. State and local 
public schools are not isolated systems, but are 
impacted by various Federal/national initiatives, 
local constituent concerns, and educational/profes- 
sional movements. Native education involves all 
three systems of providers as well as the various 
Native stakeholder groups served through the sys- 
tems. Thus, evaluation of Native education can no 
longer look only at federal policy or programs, or 
state/local initiatives. In order to be effective, the 
evaluation of Native education must examine the 
total world of Indian education as a complex sys- 
tem of interrelationships. 

Heterarchy: Change and effect in Native 
education moves not only up and down the 
spectrum, from the federal level to the local or 
Native community level, but also responds to other 
simultaneous factors such as political and educa- 
tional/professional concerns. The evaluation of 
Native education cannot focus only on a hierarchi- 
cal ordering of federal policy and program impacts, 
but must also take into account these other factors. 
For example, the local political climate affects how 
much input Native communities have into local as 
well as federal initiatives. Similarly, research on 
how students learn, new national initiatives, the 
professionalization of Native communities also 
simultaneously affects how Native education is 
implemented. Evaluation must take into con- 
sideration all of these factors in order to be respon- 
sive and valid. 

Holographic: Native education involves the 
multiple realities of many stakeholder groups. A 
given change in one part of the system, e,g, t federal 
program regulations, will have several different 
interpretations by different stakeholders. The ac- 
curacy of each interpretation is facilitated by the 
amount of information sharing and interaction 
among the stakeholders. This sharing and inter- 
action in turn changes the respective perspectives 
of the stakeholders. Evaluation must take into 
account these evolving perspectives and changes. 
There are multiple realities of what constitutes 
problems and what constitutes appropriate solu- 
tions. These multiple perspectives must be taken 
into account. 

Indeterminate: The future of Native education 
cannot be simply determined; it must be monitored 
to ensure that all anticipated and unanticipated 
factors - inputs, outputs, sideputs are regularly 
accounted for and analyzed in order to assure a 
true picture of Native education. 

Mutual Causality: Native education does not 
involve a simple push-pull of the various forces and 
stakeholders involved. Changes in Native educa- 
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tion involve an interactive and simultaneous move- 
ment of influences. Evaluation must describe how 
th*36 inf y nces interact. 

Morphogenesis: As time passes, new educa- 
* : M f technical and administrative solutions arise 
J.re used to address Native education 
problems. These new solutions can affect the ways 
in which Native students learn as well as how 
services are delivered. Berg and Ohler (INAR Sup- 
plement Volume, Commissioned Paper 11) present 
a comprehensive discussion of the possibilities for 
Native education through accessing and utilizing 
advanced techniques. 

Perspective: The continuous evaluation of Na- 
tive education should represent the diverse values 
and points of view of the various stakeholders 
including the educational concerns of teachers 
(both Native and non-Native) of Native students, 
the cultural and social concerns of Native Elders 
and parents, as well as the political concerns of 
Native governments, and the efficiency concerns of 
the various government educational service 
providers. Decisions are never value free or 
neutral; thus, decision-making must consider the 
extent to which the competing values of th e various 
stakeholders groups can be included. 

Using the framework of the emergent paradigm 
offers the various stakeholders involved in Native 
education a systemic approach to identifying what 
is not working, determining what does work, and 
how and why certain approaches to Native educa- 
tion work. The emergent paradigm offers us a view 
of Native education much more in line with the 
intergovernmental, multiprogrammatic and cul- 
turally diverse realities and conflicting goal orien- 
tations of Native education. Instead of viewing the 
organism of Native education as chaotic and un- 
planned, the paradigm offers us a way of explain- 
ing this complex creature. 

B. A Model For Continuous 
Evaluation 

It is important to examine how complex or- 
ganizations like the Department of Education and 
BIA really function in order to see how these func- 
tions affect the way research, especially evalua 
tion, are implemented and used. Again in the 
language of the Old-New Paradigm, we must un- 
derstand that viewing the evaluation strategies 
only in terms of classic bureaucratic models (Old 
Paradigm) only serves to make any potential 
strategies ineffective. This is to say that by examin- 
ing only changes in policy, programs and program- 
matic inputs like appropriations and regulatory 
mechanisms, we are doing ourselves a disservice. 



Naturalistic inquiry or "fourth generation 91 
evaluation presents us with step-by-step strategies 
for ensuring that the interests of the various Na- 
tive groups represented at the INAR Task Force 
are dealt with equitably in future evaluation and 
research of Native education. These strategies are 
outlined in capsule form by Guba and Lincoln 
(1989); the basic components they present are; 

1. Identify the full array of stakeholders who 
have an interest in or will be impacted by 
the projected evaluation. Some of these 
have been identified in the previous sec- 
tion, however, the way must be open to the 
inclusion of any new stakeholders. 

2. Elicit from each stakeholder group its con- 
cerns and interests regarding the object of 
the evaluation and the range of claims and 
issues the group wishes to raise in relation 
to the evaluation. This process must be 
open-ended in order to guarantee that it is 
an insider's view that emerges rather than 
an outsider's view. 

3. Provide a context and a methodol- 
ogy ... through which different perspec- 
tives and different claims, concerns, and 
issues, can be understood, critiqued, and 
taken into account. 

4. Generate consensus with respect to as 
many perspectives and stakeholder 
claims, concerns, and issues, as possible. 
These attempts at gaining consensus 
should be undertaken within 
groups ... and between groups. 

5. Prepare an agenda for negotiation on items 
about which there is no, or incomplete, 
consensus. Failure to reach consensus im- 
plies the continuation of competing 
perspectives, which ... can be changed 
only through new information or increased 
sophistication. Because more information 
may be required than is possible to obtain, 
given constraints of time and/or resources, 
the evaluator must devise some 
means ... for prioritising the unresolved 
items. Stakeholder inputs are essential in 
this determination, lest this need be taken 
as an opportunity to disempower selected 
stakeholders. 

6. Collect and provide the information called 
for in the con text for negotiation. It cannot 
be guaranteed that needed information 
can be provided, but every effort must be 
made to do so. 



20 



Evaluation 



Richard Nichols 



7. Establish and mediate a forum of 
stakeholder representatives in which 
negotiations can take place. Unresolved 
differences, as well as resolved claims, con* 
corns, issues, should be reviewed in the 
light of the new information in the hope 
that their number can be considerably 
reduced. Some items will likely remain 
unresolved, thereby setting the stage for 
later rounds of evaluation. Outcomes of 
this step must include definitive actions if 
the negotiation is to be regarded as sue* 
cessful. 

8. Develop a report, probably several reports, 
that communicates to each stakeholder 
group any consensus on perspectives and 
findings and any resolution regarding 
claims, concerns, and issues that they or 
other groups have raised. 

9. Recycle the evaluation once again to take 
up still unresolved perspectives and find- 
ings and questions and their related 
claims, concerns, and issues. New aspects 
may also be explored as they emerge as a 
result of the first-round evaluation. Fourth 
generation evaluations are never com- 
pleted. 

Native education is a complex organization, 
involving several stakeholders - the many systems 
of educational service providers and the Native 
students, parents and Elders served. Each of these 
stakeholder groups has differing perceptions about 
the goals of Native education as well as the best 
ways in which to meet those goals. The emerging 
paradigm and the fourth generation evaluation 
model recognize the diversity that the world of 
Native education represents. 

Policy making within this world involves 
several competing goals and priorities. Adaptive 
program implementation strategies are the most 
appropriate for organizational systems with con- 
flicting goals and where many stakeholder groups 
are involved. Within the world of Native educa- 
tion, many adaptive strategies can be brought to 
bear on creating change. Beaulieu (INAR Supple- 
ment Volume, Commissioned Paper 20) presents 
strategies on how public education can be com- 
pelled to be more responsive to Native students 
needs and the goals of Native education. Similar- 
ly, in evaluation, recognition of the disparate goals 
of Native stakeholders and those who provide for 
Native education can be accommodated through 
the use of the naturalistic inquiry or fourth genera- 
tion strategies. Such evaluation strategies can 
more accurately reflect the various perspectives 



and give us a truer and thus, more useful picture 
of Native education. 

Naturalistic evaluation is designed to resolve 
conflicting claims, concerns and issues and to move 
stakeholders toward a consensus of critical issues. 
Naturalistic evaluation methods rely heavily on 
the collection and dissemination of information as 
descriptions of experiences related to key claims, 
concerns and issues relevant to as many 
stakeholder groups as possible. As noted earlier, 
the case study approach is an important element 
of naturalistic evaluation. 

Regarding the use of the case study as an 
evaluation approach particularly appropriate for 
Native education, this author would suggest that 
the reader obtain a copy of the Fall 1981 issue of 
Daedalus: Journal of American Academy of Arts 
and Sciences. The issue, entitled "America's 
Schools: Portraits and Perspectives/ 1 presents 
case studies of several different schools repre- 
sentative of (1) the range of schools attended by 
American students, and (2) particular types of 
schools serving particular subgroups in American 
society. Similar descriptive studies of Native 
education programs, claims and concerns w mid be 
a step forward in presenting evaluation as useful 
to Native stakeholders. 

The use of the naturalistic evaluation model 
would allow for the continuous exchange of infor- 
mation, reactions and responses to the cycles of 
information dissemination. The process results in 
awareness of changes, perspectives and move- 
ments toward consensus. The INAR hearings have 
served as one cycle of information gathering; the 
INAR Final Report and individual commissioned 
papers in the Supplement Volume serve as an 
analysis step in another cycle; and the dissemina- 
tion of the INAR report will serve as another cycle 
of continuing movement toward resolution of con- 
flicting Native education goals. In the future, the 
White House Conference on Indian Education can 
serve as another cycle in the evaluation process. 
The continuing communication and review of infor- 
mation is critical. The future willingness of the 
various providers of Native education to be respon- 
sive to these perspectives is critical. Perhaps, the 
INAR report as well as the White House Con- 
ference on Indian Education will result in the pub- 
lication of more research on "what works" in Native 
education, just as the A Nation At Risk report 
generated several more reports and evaluation ef- 
forts. 

Conclusions 

The many studies that have documented 
changing policies and programs in Native educa- 
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tion have had positive impacts albeit in some cases 
incremental, insufficient or inadequate. The INAR 
report provides an opportunity for furthering a 
truly representative Native view of what needs to 
be done to improve Native education. By directing 
further evaluation efforts on specific cases of what 
works and focusing on how and why "it" works, we 
can ensure that the INAR report is not viewed by 
Native people as "just another study." 

By providing for continuing evaluations that 
will be able to (1) address a number of stakeholder 
groups, (2) provide appropriate formats for report- 
ing, and (3) facilitate effective exchange of data and 
findings among Native Elders, educators, parents, 
and leaders, we can perhaps begin to form consen- 
sus on a new direction for Native education. 

(The author wishes to acknowledge Gwen 
Shunatona and Anne Litchfield for their contribu- 
tion of ideas and their editorial input and sugges- 
tions which greatly facilitated the writing of this 
document.) 
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